Utilization of reflectivity to determine changes to traffic infrastructure elements

ABSTRACT

Systems, methods, and computer-readable media are provided for analyzing a traffic infrastructure element, determining reflectivity information of the traffic infrastructure element based on the analyzing of the traffic infrastructure element, comparing the reflectivity information of the traffic infrastructure element with semantic map information of the traffic infrastructure element, and providing instructions to an autonomous vehicle based on the comparing of the reflectivity information of the traffic infrastructure element with the semantic map information of the traffic infrastructure element.

BACKGROUND 1. Technical Field

The subject technology provides solutions for roadway infrastructureelements, and in particular, for determining changes in roadwayinfrastructure elements based on reflectivity.

2. Introduction

When operating an autonomous vehicle fleet, the health and utilizationof the traffic infrastructure is important. Degraded trafficinfrastructure affects service quality and can lead to disruption andhigher accident rates, thereby causing the autonomous vehicle to deviatefrom its intended path, causing damage to the autonomous vehicle, orcausing a degradation in rider safety and comfort. For example, missingstop signs can compromise the ability of the autonomous vehicle tonavigate intersections safely as this may cause the other road actors'behaviors more unpredictable. In general, poor/missing infrastructureelements (e.g., poor/missing signage lane markings, etc.) make it moredifficult to navigate, which causes traffic jams, impacts trip time andservice quality, and compromises safety.

Moreover, there is no centralized unit such as a city that maintains thetraffic infrastructure as each instance of degradation is handled on acase-by-case basis as they are reported. A complete picture of the stateof the traffic infrastructure including degraded elements (e.g.,missing/vandalized signage) and problematic areas (e.g., frequentlycongested areas) is of high value to any organization such as a city asa planning tool and a means to prioritize construction efforts.

As such, a need exists for a system and a method that can efficientlyand effectively detect changes in roadway infrastructure elements basedon reflectivity, thereby allowing for rapid and safe routedetermination.

BRIEF DESCRIPTION OF THE DRAWINGS

Certain features of the subject technology are set forth in the appendedclaims. However, the accompanying drawings, which are included toprovide further understanding, illustrate disclosed aspects and togetherwith the description serve to explain the principles of the subjecttechnology. In the drawings:

FIG. 1 illustrates an example environment that includes an autonomousvehicle in communication with a remote computing system, according tosome aspects of the disclosed technology.

FIG. 2 illustrates an example flow chart of a degrading trafficinfrastructure system, according to some aspects of the disclosedtechnology.

FIGS. 3A and 3B illustrate example road signage, according to someaspects of the disclosed technology.

FIG. 4 illustrates an example flow chart of a reflectivity utilizationprocess, according to some aspects of the disclosed technology.

FIG. 5 illustrates an example process for utilizing reflectivity todetermine changes to a traffic infrastructure element, according to someaspects of the disclosed technology.

FIG. 6 illustrates an example processor-based system with which someaspects of the subject technology can be implemented.

DETAILED DESCRIPTION

Various examples of the present technology are discussed in detailbelow. While specific implementations are discussed, it should beunderstood that this is done for illustration purposes only. A personskilled in the relevant art will recognize that other components andconfigurations may be used without parting from the spirit and scope ofthe present technology. In some instances, well-known structures anddevices are shown in block diagram form in order to facilitatedescribing one or more aspects. Further, it is to be understood thatfunctionality that is described as being carried out by certain systemcomponents may be performed by more or fewer components than shown.

FIG. 1 illustrates an example autonomous vehicle environment 100. Theexample autonomous vehicle environment 100 includes an autonomousvehicle 102, a remote computing system 150, and a ridesharingapplication 170. The autonomous vehicle 102, remote computing system150, and ridesharing application 170 can communicate with each otherover one or more networks, such as a public network (e.g., a publiccloud, the Internet, etc.), a private network (e.g., a local areanetwork, a private cloud, a virtual private network, etc.), and/or ahybrid network (e.g., a multi-cloud or hybrid cloud network, etc.).

The autonomous vehicle 102 can navigate about roadways without a humandriver based on sensor signals generated by sensors 104-108 on theautonomous vehicle 102. The sensors 104-108 on the autonomous vehicle102 can include one or more types of sensors and can be arranged aboutthe autonomous vehicle 102. For example, the sensors 104-108 caninclude, without limitation, one or more inertial measuring units(IMUs), one or more image sensors (e.g., visible light image sensors,infrared image sensors, video camera sensors, surround view camerasensors, etc.), one or more light emitting sensors, one or more globalpositioning system (GPS) devices, one or more radars, one or more lightdetection and ranging sensors (LIDARs), one or more sonars, one or moreaccelerometers, one or more gyroscopes, one or more magnetometers, oneor more altimeters, one or more tilt sensors, one or more motiondetection sensors, one or more light sensors, one or more audio sensors,etc. In some implementations, sensor 104 can be a radar, sensor 106 canbe a first image sensor (e.g., a visible light camera), and sensor 108can be a second image sensor (e.g., a thermal camera). Otherimplementations can include any other number and type of sensors.

The autonomous vehicle 102 can include several mechanical systems thatare used to effectuate motion of the autonomous vehicle 102. Forinstance, the mechanical systems can include, but are not limited to, avehicle propulsion system 130, a braking system 132, and a steeringsystem 134. The vehicle propulsion system 130 can include an electricmotor, an internal combustion engine, or both. The braking system 132can include an engine brake, brake pads, actuators, and/or any othersuitable componentry configured to assist in decelerating the autonomousvehicle 102. The steering system 134 includes suitable componentryconfigured to control the direction of movement of the autonomousvehicle 102 during navigation.

The autonomous vehicle 102 can include a safety system 136. The safetysystem 136 can include lights and signal indicators, a parking brake,airbags, etc. The autonomous vehicle 102 can also include a cabin system138, which can include cabin temperature control systems, in-cabinentertainment systems, etc.

The autonomous vehicle 102 can include an internal computing system 110in communication with the sensors 104-108 and the systems 130, 132, 134,136, and 138. The internal computing system 110 includes one or moreprocessors and at least one memory for storing instructions executableby the one or more processors. The computer-executable instructions canmake up one or more services for controlling the autonomous vehicle 102,communicating with remote computing system 150, receiving inputs frompassengers or human co-pilots, logging metrics regarding data collectedby sensors 104-108 and human co-pilots, etc.

The internal computing system 110 can include a control service 112configured to control operation of the vehicle propulsion system 130,the braking system 132, the steering system 134, the safety system 136,and the cabin system 138. The control service 112 can receive sensorsignals from the sensors 104-108 can communicate with other services ofthe internal computing system 110 to effectuate operation of theautonomous vehicle 102. In some examples, control service 112 may carryout operations in concert with one or more other systems of autonomousvehicle 102.

The internal computing system 110 can also include a constraint service114 to facilitate safe propulsion of the autonomous vehicle 102. Theconstraint service 116 includes instructions for activating a constraintbased on a rule-based restriction upon operation of the autonomousvehicle 102. For example, the constraint may be a restriction onnavigation that is activated in accordance with protocols configured toavoid occupying the same space as other objects, abide by traffic laws,circumvent avoidance areas, etc. In some examples, the constraintservice 114 can be part of the control service 112.

The internal computing system 110 can also include a communicationservice 116. The communication service 116 can include software and/orhardware elements for transmitting and receiving signals to and from theremote computing system 150. The communication service 116 can beconfigured to transmit information wirelessly over a network, forexample, through an antenna array or interface that provides cellular(long-term evolution (LTE), 3rd Generation (3G), 5th Generation (5G),etc.) communication.

In some examples, one or more services of the internal computing system110 are configured to send and receive communications to remotecomputing system 150 for reporting data for training and evaluatingmachine learning algorithms, requesting assistance from remote computingsystem 150 or a human operator via remote computing system 150, softwareservice updates, ridesharing pickup and drop off instructions, etc.

The internal computing system 110 can also include a latency service118. The latency service 118 can utilize timestamps on communications toand from the remote computing system 150 to determine if a communicationhas been received from the remote computing system 150 in time to beuseful. For example, when a service of the internal computing system 110requests feedback from remote computing system 150 on a time-sensitiveprocess, the latency service 118 can determine if a response was timelyreceived from remote computing system 150, as information can quicklybecome too stale to be actionable. When the latency service 118determines that a response has not been received within a thresholdperiod of time, the latency service 118 can enable other systems ofautonomous vehicle 102 or a passenger to make decisions or provideneeded feedback.

The internal computing system 110 can also include a user interfaceservice 120 that can communicate with cabin system 138 to provideinformation or receive information to a human co-pilot or passenger. Insome examples, a human co-pilot or passenger can be asked or requestedto evaluate and override a constraint from constraint service 114. Inother examples, the human co-pilot or passenger may wish to provide aninstruction to the autonomous vehicle 102 regarding destinations,requested routes, or other requested operations.

As described above, the remote computing system 150 can be configured tosend and receive signals to and from the autonomous vehicle 102. Thesignals can include, for example and without limitation, data reportedfor training and evaluating services such as machine learning services,data for requesting assistance from remote computing system 150 or ahuman operator, software service updates, rideshare pickup and drop offinstructions, etc.

The remote computing system 150 can include an analysis service 152configured to receive data from autonomous vehicle 102 and analyze thedata to train or evaluate machine learning algorithms for operating theautonomous vehicle 102. The analysis service 152 can also performanalysis pertaining to data associated with one or more errors orconstraints reported by autonomous vehicle 102.

The remote computing system 150 can also include a user interfaceservice 154 configured to present metrics, video, images, soundsreported from the autonomous vehicle 102 to an operator of remotecomputing system 150, maps, routes, navigation data, notifications, userdata, vehicle data, software data, and/or any other content. Userinterface service 154 can receive, from an operator, input instructionsfor the autonomous vehicle 102.

The remote computing system 150 can also include an instruction service156 for sending instructions regarding the operation of the autonomousvehicle 102. For example, in response to an output of the analysisservice 152 or user interface service 154, instructions service 156 canprepare instructions to one or more services of the autonomous vehicle102 or a co-pilot or passenger of the autonomous vehicle 102.

The remote computing system 150 can also include a rideshare service 158configured to interact with ridesharing applications 170 operating oncomputing devices, such as tablet computers, laptop computers,smartphones, head-mounted displays (HMDs), gaming systems, servers,smart devices, smart wearables, and/or any other computing devices. Insome cases, such computing devices can be passenger computing devices.The rideshare service 158 can receive from passenger ridesharing app 170requests, such as user requests to be picked up or dropped off, and candispatch autonomous vehicle 102 for a requested trip.

The rideshare service 158 can also act as an intermediary between theridesharing app 170 and the autonomous vehicle 102. For example,rideshare service 158 can receive from a passenger instructions for theautonomous vehicle 102, such as instructions to go around an obstacle,change routes, honk the horn, etc. The rideshare service 158 can providesuch instructions to the autonomous vehicle 102 as requested.

The remote computing system 150 can also include a package service 162configured to interact with the ridesharing application 170 and/or adelivery service 172 of the ridesharing application 170. A useroperating ridesharing application 170 can interact with the deliveryservice 172 to specify information regarding a package to be deliveredusing the autonomous vehicle 102. The specified information can include,for example and without limitation, package dimensions, a packageweight, a destination address, delivery instructions (e.g., a deliverytime, a delivery note, a delivery constraint, etc.), and so forth.

The package service 162 can interact with the delivery service 172 toprovide a package identifier to the user for package labeling andtracking. Package delivery service 172 can also inform a user of whereto bring their labeled package for drop off. In some examples, a usercan request the autonomous vehicle 102 come to a specific location, suchas the user's location, to pick up the package. While delivery service172 has been shown as part of the ridesharing application 170, it willbe appreciated by those of ordinary skill in the art that deliveryservice 172 can be its own separate application.

One beneficial aspect of utilizing autonomous vehicle 102 for bothridesharing and package delivery is increased utilization of theautonomous vehicle 102. Instruction service 156 can continuously keepthe autonomous vehicle 102 engaged in a productive itinerary betweenrideshare trips by filling what otherwise would have been idle time withproductive package delivery trips.

When operating an autonomous vehicle fleet, the health and utilizationof the traffic infrastructure is important. Degraded trafficinfrastructure affects service quality and can lead to disruption andhigher accident rates, thereby causing the autonomous vehicle to deviatefrom its intended path, causing damage to the autonomous vehicle, orcausing a degradation in rider safety and comfort. For example, missingstop signs can compromise the ability of the autonomous vehicle tonavigate intersections safely as this may cause the other road actors'behaviors more unpredictable. In general, poor/missing infrastructureelements (e.g., poor/missing signage lane markings, etc.) make it moredifficult to navigate, which causes traffic jams, impacts trip time andservice quality, and compromises safety.

Moreover, there is no centralized unit such as a city that maintains thetraffic infrastructure as each instance of degradation is handled on acase-by-case basis as they are reported. A complete picture of the stateof the traffic infrastructure including degraded elements (e.g.,missing/vandalized signage) and problematic areas (e.g., frequentlycongested areas) is of high value to any organization such as a city asa planning tool and a means to prioritize construction efforts.

As such, a need exists for a system and a method that can efficientlyand effectively detect changes in roadway infrastructure elements basedon reflectivity, thereby allowing for rapid and safe routedetermination.

FIG. 2 illustrates an example flow chart of a degrading trafficinfrastructure system 200, according to some aspects of the disclosedtechnology. In some instances, the degrading traffic infrastructuresystem 200 can include a cloud application programming interface (API)202, customers 204, an autonomous vehicle fleet network 206, andautonomous vehicles 208. In some cases, the degrading trafficinfrastructure system 200 can assist the autonomous vehicle fleet 208avoid areas with degraded traffic infrastructure. In other instances,the degrading traffic infrastructure system 200 can automate compilingof traffic infrastructure information and maintain a dynamic overallpicture of a city's traffic infrastructure including temporary signagesand construction. The degrading traffic infrastructure system 200 canallow the traffic infrastructure information to be more consistent,objective, and increase the frequency in which the quality of thetraffic infrastructures are assessed (e.g., “24/7”).

The degrading traffic infrastructure system 200 can also implement atraffic infrastructure monitoring system (e.g., with the autonomousvehicles 208) that includes sensors (e.g., inertial measurement unit(IMU), wheel encoders, light detection and ranging (LiDAR), radar, andcameras in the autonomous vehicles 208). The degrading trafficinfrastructure system 200 can further compute capabilities, potentiallyadding additional sensors (infrared (IR), ultrasonic, etc.) andcomputational devices. Data processing can be performed in the cloud 202or external servers to divert additional computational resources fromthe autonomous vehicle computes. The sensors can be utilized todetermine reflectivity of traffic infrastructure elements such astraffic signs and road signage.

In other instances, the degrading traffic infrastructure system 200 canutilize a sensor suite of LiDAR, IMUs, cameras, and radar that canprovide a significant amount of information to infer and quantify thehealth of the traffic infrastructure such as reflectivity of trafficsigns/road signage. For example, traffic infrastructure data of roadsignage can be collected via the sensor suite placed around the vehiclesuch as a downward pointing LIDAR, RADAR, camera, IR, ultrasonicdistance detection, and IR thermal cameras.

In some instances, the degrading traffic infrastructure system 200 cancollect and provide road and infrastructure data collection to improveride comfort and to update blacklists to provide ride efficiency,distance, and duration. Moreover, traffic light reliability and trafficsign confusion or disappearance can be provided to third parties (e.g.,cities, transportation companies, and infrastructure designers) toexecute repairs, or via an API or an application (“app”) for road users.The degrading traffic infrastructure system 200 also can conduct trafficusability, infrastructure and layout utilization, and safety studieswhile leveraging the data.

The degrading traffic infrastructure system 200 can assess and predicttraffic infrastructure health, while also recommending trafficinfrastructure improvements using the sensory, computational, andcommunication capabilities of a fleet of autonomous vehicles 208 overthe operation domain/network 206. For example, reflectivity of trafficsigns can be assessed to predict the “health” of the traffic sign,thereby providing data that may recommend that the traffic sign bereplaced, improved, fixed, etc.

In some instances, the autonomous vehicle fleet 208 can include theunique advantage of traversing the streets of a city, state, or nationon a daily basis. In other instances, the autonomous vehicle fleet 208can utilize high fidelity LIDAR point clouds for localization andsemantic map information aggregation in a mapping stack, which can allowfor a baseline to track and compare various components of the degradingtraffic infrastructure system 200.

The autonomous vehicle fleet 208 can be employed to obtain detailedinformation on traffic infrastructure including traffic signs, lanemarkings, traffic signals, and any other traffic infrastructure suitablefor the intended purpose and understood by a person of ordinary skill inthe art. In some instances, the autonomous vehicle fleet 208 can alsoemploy sensor sets, mathematical algorithms for interpreting data frommultiple sensors, and additional measurement devices to assist indetermining the reflectivity traffic infrastructure state.

In some instances, the degrading traffic infrastructure system 200 cancollect traffic infrastructure data from the autonomous vehicle fleet208, which can be communicated to the cloud 202. The cloud 202 can thenprocess and provide the traffic infrastructure data, in-real time whereapplicable, via an API. The processed traffic infrastructure data can becommunicated throughout the autonomous vehicle fleet 208 to inform andfacilitate regular operation of the autonomous vehicles 208. Forexample, the degrading traffic infrastructure system 200 can includelabeling and blacklisting road segments with degraded infrastructure(e.g., construction sites, missing signs, vandalized traffic signs,etc.) and routing based on a road utilization quality index (e.g.,avoiding poorly signalized intersections at high traffic times).

The sensor data collected from the autonomous vehicles 208 can beutilized to compute metrics describing various aspects of the state ofspecific road segments. For example, a traffic infrastructurescore/index for each road segment can be calculated by analyzingdifferences between different LiDAR scans or camera images over time andanalyzing times and locations where the autonomous vehicles 208 havedifficulty navigating. In some instances, specific types and modes ofdegradation in the traffic infrastructure can create a hazardous drivingcondition (e.g., missing/vandalized stop signs) can be annotated on amap used for coordinating fleet operation.

In some instances, the degrading traffic infrastructure system 200 caninclude examining the validity of current infrastructure (e.g., missingor damaged signage signals, and road markings that are incorrect orprovide insufficient information) and analyzing the impact of affectingchanges (e.g., closing a street, changing a speed limit, and whichinfrastructure elements may be tied to these changes). In otherinstances, the degrading traffic infrastructure system 200 can include acomparison of a current map and a semantic understanding of the city(e.g., whether traffic signs are in the correct location, whether lanemarkings are correctly placed, and whether traffic is being poorlydirected). In some instances, the degrading traffic infrastructuresystem 200 can further utilize infrastructure visibility (e.g., lanemarking visibility, sign disappearance or degradation, and traffic lightoutage) and roadway usage (e.g., whether other road users are utilizingthe road correctly), which can provide insights into reflectivity,degradation, usability, and overall validity of the trafficinfrastructure.

In other instances, the degrading traffic infrastructure system 200 caninclude high resolution mapping that can provide pinpoint roadinformation in space and time. The degrading traffic infrastructuresystem 200 can further communicate the traffic infrastructure data tothe autonomous vehicles 208 and identify potential hazards at keylocations along their respective routes.

Reflectivity:

In some instances, the degrading traffic infrastructure system 200 candetect radar, LiDAR, and/or visual reflectivity/return intensitychanges, some of which include various roadway infrastructure elementsthat can provide key insights as to their health, legibility, andvalidity of placement. For example, radar reflectivity can provideinformation regarding material changes to road signage such asdifferences between glass, fog, plastic, and various metals. In someinstances, the degrading traffic infrastructure system 200 can detectwhen the material of a road sign has changed. For example, a metal signthat has been covered with a plastic sign/cover may be detected and thedegrading traffic infrastructure system 200 may then provide a signal ordetermination that the road sign has been vandalized or tampered with.

LiDAR reflectivity changes can occur at smaller LiDAR wavelengths andcan provide information regarding aspect and surface treatment ofmaterials. For example, utilizing LiDAR reflectivity, the degradingtraffic infrastructure system 200 can detect whether a retro-reflectivesign has been tampered with and whether a sign has deteriorated bycomparing the difference between mapped intensity data and surroundingintensity data that can include highly dense and accumulated real-timescans.

In some instances, reflectivity information of traffic infrastructureelements can include reflectivity information of the whole or a portionof the traffic infrastructure element. For example, the whole trafficinfrastructure element can include the reflective red portion of a stopsign, while a portion of the traffic infrastructure element can includethe reflective numbers of a speed limit sign.

As shown in FIGS. 3A and 3B, FIG. 3A illustrates an untampered roadsignage 302 of 10 MPH, while FIG. 3B illustrates a tampered road signage304 of 70 MPH. In this example, the road signage has been vandalized byplacement of reflective tape 306 to change “1” to “7.” Under normalcircumstances, an autonomous vehicle would detect the speed limitsignage and increase the speed of the autonomous vehicle accordingly.However, this can lead to a very dangerous situation where theautonomous vehicle dictates its speed solely on the speed limit postedon the road signage. In such a case, as in FIGS. 3A and 3B, theautonomous vehicle may suddenly increase its speed to 70 MPH, when thesurrounding vehicles are only traveling at 10 MPH, potentially causing acollision. However, in some embodiments as described herein, theautonomous vehicle's behavior may be based on the correctly identifiedspeed limit or traffic sign (e.g., by maintaining the speed asdesignated by a semantic map), in the event that the autonomous vehicledetects that the traffic sign has been tampered with (e.g., measuredreflectivity parameters do not match those of the semantic map). In thecase, as shown in FIGS. 3A and 3B, the autonomous vehicle may drive at10 MPH, even though the traffic sign appears to state 70 MPH, based onthe reflectivity comparison.

Utilizing changes in reflectivity by the degrading trafficinfrastructure system 200 can also be applied to added, removed, ormodified road signage. For example, these “changed” road signages willprovide different LiDAR intensity/reflective data as compared to theoriginal road signage reflective data. When the degrading trafficinfrastructure system 200 detects a change in reflectivity based on theroad signage, the road signage can be flagged or placed on a blacklistfor review.

Changes in color of signals or other signage can also be detected by thedegrading traffic infrastructure system 200 by utilizing the visiblespectrum, close ultraviolet (UV), and infrared (IR). The degradingtraffic infrastructure system 200 can provide 2D information on themodification of visible colors of infrastructure elements. The degradingtraffic infrastructure system 200 can assist in determining changes inother road users' behaviors, as well as flagging degradations orpossible future degradations of visibility (e.g., by human eyes) ordetectability (e.g., by sensors and algorithms) of such signage. In someinstances, the degrading traffic infrastructure system 200 can furtherdetermine when traffic lights have changed colors (e.g., the trafficlight bulbs were replaced for a different colored bulb).

FIG. 4 illustrates an example flow chart of a reflectivity utilizationprocess 400, according to some aspects of the disclosed technology.

In some instances, the reflectivity utilization process 400 can includeautonomous vehicles encountering a road sign, traffic light, roadmarker, traffic controls, constructions zones, or any other trafficinfrastructure element suitable for the intended purpose and understoodby a person of ordinary skill in the art 402. The reflectivityutilization process 400 can further include obtaining and processingsensory information regarding the road sign 404 (e.g., 3D LiDAR pointcloud, reflectivity, visual attributes, and sign classification).

In some instances, the reflectivity utilization process 400 can includecomparing the sensory information from step 404 with semantic mapinformation 406 (e.g., whether there is an expected road sign at anintersection). If the sensory information 404 does not match 408 withthe semantic map information 406, the change or difference is loggedaccordingly and a corresponding action is taken such as calling forremote assistance from an operator. If the sensory information 404 doesmatch 408 with the semantic map information 406, the reflectivityutilization process 400 can include comparing the semantic mapinformation 406 with reflectivity and visual information 412.

If the reflectivity and visual information 412 does not match 414 withthe semantic map information 406, the change or difference is loggedaccordingly and a corresponding action is taken such as calling forremote assistance from an operator. If the reflectivity and visualinformation 412 does match 414 with the semantic map information 406,the reflectivity utilization process 400 can include comparing thesemantic map information 406 with 3D shape information 418.

If the 3D shape information 418 does not match 420 with the semantic mapinformation 406, the change or difference is logged accordingly and acorresponding action is taken such as calling for remote assistance froman operator. If 3D shape information 418 does match 420 with thesemantic map information 406, the reflectivity utilization process 400can include instructions for the autonomous vehicle to proceed asnormal. The 3D shape information 418 of the road sign can include theexterior 3D shape information that can indicate a particular type ofroad sign.

In some instances, the reflectivity utilization process 400 can furtherinclude repeating steps 402-424 for a plurality of autonomous vehicles(e.g., from an autonomous vehicle fleet) to provide confirmation of achange in reflectivity of the traffic infrastructure element (e.g., aroad sign). For example, the autonomous vehicle fleet can providerespective reflectivity measurements of the same traffic infrastructureelement to a backend network to determine whether a change (e.g.,physical change) has occurred to the traffic infrastructure element. Ifthe reflectivity measurements of the respective autonomous vehicles donot match, then the semantic maps may not yet be updated or reviseduntil the reported change of the traffic infrastructure element has beenconfirmed. If the reflectivity measurements of the respective autonomousvehicles do match, then the semantic maps may be updated or revisedaccordingly and provided to the autonomous vehicle fleet for use.

Having disclosed some example system components and concepts, thedisclosure now turns to FIG. 5, which illustrate example method 500 forutilizing reflectivity to determine changes to a traffic infrastructureelement. The steps outlined herein are exemplary and can be implementedin any combination thereof, including combinations that exclude, add, ormodify certain steps.

At step 502, the method 500 can include analyzing a trafficinfrastructure element. In some instances, the traffic infrastructureelement can be a road sign, a traffic light, signage, or any othertraffic infrastructure element suitable for the intended purpose andunderstood by a person of ordinary skill in the art.

At step 504, the method 500 can include determining reflectivityinformation of the traffic infrastructure element based on the analyzingof the traffic infrastructure element. In some instances, thereflectivity information of the traffic infrastructure element can be a3D light detection and ranging (LiDAR) point cloud. In other instances,the reflectivity information of the traffic infrastructure element canbe radar information or visual reflectivity/return intensity changeinformation. The method 500 can utilize LIDAR, RADAR, camera, IR,ultrasonic distance detection, or IR thermal cameras to determine thereflectivity information of the traffic infrastructure element.

In some instances, the reflectivity information of the trafficinfrastructure element can include reflectivity information of the wholeor a portion of the traffic infrastructure element. For example, thewhole traffic infrastructure element can include the reflective redportion of a stop sign, while a portion of the traffic infrastructureelement can include the reflective numbers of a speed limit sign.

At step 506, the method 500 can include comparing the reflectivityinformation of the traffic infrastructure element with semantic mapinformation of the traffic infrastructure element. In some instances,the semantic map information of the traffic infrastructure element caninclude previous reflectivity information of the traffic infrastructureelement.

At step 506, the method 500 can include providing instructions to anautonomous vehicle based on the comparing of the reflectivityinformation of the traffic infrastructure element with the semantic mapinformation of the traffic infrastructure element. In some instances,the providing of the instructions to the autonomous vehicle can includelogging a difference between the reflectivity information of the trafficinfrastructure element and the semantic map information of the trafficinfrastructure element, or proceeding as normal.

The method 500 can further include comparing visual information of thetraffic infrastructure element with the semantic map information of thetraffic infrastructure element. In some instances, the visualinformation of the traffic infrastructure element can include colorinformation of the traffic infrastructure element. In other instances,the visual information of the traffic infrastructure element can bedetermined by utilizing the visible spectrum, close ultraviolet (UV), orinfrared (IR).

The method 500 can further include comparing 3D shape information of thetraffic infrastructure element with the semantic map information of thetraffic infrastructure element. In some instances, the 3D shapeinformation of the traffic infrastructure element can include a type,style, composition, or any other 3D shape information suitable for theintended purpose and understood by a person of ordinary skill in theart.

In some instances, the method 500 can further include repeating steps502-508 for a plurality of autonomous vehicles (e.g., from an autonomousvehicle fleet) to provide confirmation of a change in reflectivity ofthe traffic infrastructure element (e.g., a road sign). For example, themethod 500 can include the autonomous vehicle fleet providing respectivereflectivity measurements of the same traffic infrastructure element toa backend network to determine whether a change (e.g., physical change)has occurred to the traffic infrastructure element. If the reflectivitymeasurements of the respective autonomous vehicles do not match, thenthe method 500 may not include updating or revising the semantic mapsuntil the reported change of the traffic infrastructure element has beenconfirmed. If the reflectivity measurements of the respective autonomousvehicles do match, then the method 500 may include updating or revisingthe semantic maps accordingly and providing the updated semantic maps tothe autonomous vehicle fleet for use.

FIG. 6 illustrates an example computing system 600 which can be, forexample, any computing device making up internal computing system 110,remote computing system 150, a passenger device executing rideshareapplication 170, or any other computing device. In FIG. 6, thecomponents of the computing system 600 are in communication with eachother using connection 605. Connection 605 can be a physical connectionvia a bus, or a direct connection into processor 610, such as in achipset architecture. Connection 605 can also be a virtual connection,networked connection, or logical connection.

In some embodiments, computing system 600 is a distributed system inwhich the functions described in this disclosure can be distributedwithin a datacenter, multiple data centers, a peer network, etc. In someembodiments, one or more of the described system components representsmany such components each performing some or all of the function forwhich the component is described. In some embodiments, the componentscan be physical or virtual devices.

Example system 600 includes at least one processing unit (CPU orprocessor) 610 and connection 605 that couples various system componentsincluding system memory 615, such as read-only memory (ROM) 620 andrandom access memory (RAM) 625 to processor 610. Computing system 600can include a cache of high-speed memory 612 connected directly with, inclose proximity to, or integrated as part of processor 610.

Processor 610 can include any general purpose processor and a hardwareservice or software service, such as services 632, 634, and 636 storedin storage device 630, configured to control processor 610 as well as aspecial-purpose processor where software instructions are incorporatedinto the actual processor design. Processor 610 may essentially be acompletely self-contained computing system, containing multiple cores orprocessors, a bus, memory controller, cache, etc. A multi-core processormay be symmetric or asymmetric.

To enable user interaction, computing system 600 includes an inputdevice 645, which can represent any number of input mechanisms, such asa microphone for speech, a touch-sensitive screen for gesture orgraphical input, keyboard, mouse, motion input, speech, etc. Computingsystem 600 can also include output device 635, which can be one or moreof a number of output mechanisms known to those of skill in the art. Insome instances, multimodal systems can enable a user to provide multipletypes of input/output to communicate with computing system 600.Computing system 600 can include communications interface 640, which cangenerally govern and manage the user input and system output. There isno restriction on operating on any particular hardware arrangement, andtherefore the basic features here may easily be substituted for improvedhardware or firmware arrangements as they are developed.

Storage device 630 can be a non-volatile memory device and can be a harddisk or other types of computer readable media which can store data thatare accessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memories (RAMs), read-only memory (ROM), and/or somecombination of these devices.

The storage device 630 can include software services, servers, services,etc., that when the code that defines such software is executed by theprocessor 610, it causes the system to perform a function. In someembodiments, a hardware service that performs a particular function caninclude the software component stored in a computer-readable medium inconnection with the necessary hardware components, such as processor610, connection 605, output device 635, etc., to carry out the function.

For clarity of explanation, in some instances, the present technologymay be presented as including individual functional blocks includingfunctional blocks comprising devices, device components, steps orroutines in a method embodied in software, or combinations of hardwareand software.

Any of the steps, operations, functions, or processes described hereinmay be performed or implemented by a combination of hardware andsoftware services or services, alone or in combination with otherdevices. In some embodiments, a service can be software that resides inmemory of a client device and/or one or more servers of a contentmanagement system and perform one or more functions when a processorexecutes the software associated with the service. In some embodiments,a service is a program or a collection of programs that carry out aspecific function. In some embodiments, a service can be considered aserver. The memory can be a non-transitory computer-readable medium.

In some embodiments, the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implementedusing computer-executable instructions that are stored or otherwiseavailable from computer-readable media. Such instructions can comprise,for example, instructions and data which cause or otherwise configure ageneral purpose computer, special purpose computer, or special purposeprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The executable computer instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware, orsource code. Examples of computer-readable media that may be used tostore instructions, information used, and/or information created duringmethods according to described examples include magnetic or opticaldisks, solid-state memory devices, flash memory, USB devices providedwith non-volatile memory, networked storage devices, and so on.

Devices implementing methods according to these disclosures can comprisehardware, firmware and/or software, and can take any of a variety ofform factors. Typical examples of such form factors include servers,laptops, smartphones, small form factor personal computers, personaldigital assistants, and so on. The functionality described herein alsocan be embodied in peripherals or add-in cards. Such functionality canalso be implemented on a circuit board among different chips ordifferent processes executing in a single device, by way of furtherexample.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are means for providing the functions described inthese disclosures.

Although a variety of examples and other information was used to explainaspects within the scope of the appended claims, no limitation of theclaims should be implied based on particular features or arrangements insuch examples, as one of ordinary skill would be able to use theseexamples to derive a wide variety of implementations. Further andalthough some subject matter may have been described in languagespecific to examples of structural features and/or method steps, it isto be understood that the subject matter defined in the appended claimsis not necessarily limited to these described features or acts. Forexample, such functionality can be distributed differently or performedin components other than those identified herein. Rather, the describedfeatures and steps are disclosed as examples of components of systemsand methods within the scope of the appended claims.

Claim language reciting “at least one of” a set indicates that onemember of the set or multiple members of the set satisfy the claim. Forexample, claim language reciting “at least one of A and B” means A, B,or A and B.

What is claimed is:
 1. A computer-implemented method comprising:analyzing a traffic infrastructure element; determining reflectivityinformation of the traffic infrastructure element based on the analyzingof the traffic infrastructure element; comparing the reflectivityinformation of the traffic infrastructure element with semantic mapinformation of the traffic infrastructure element; and providinginstructions to an autonomous vehicle based on the comparing of thereflectivity information of the traffic infrastructure element with thesemantic map information of the traffic infrastructure element.
 2. Thecomputer-implemented method of claim 1, wherein the trafficinfrastructure element is a road sign, a traffic light, or signage. 3.The computer-implemented method of claim 1, wherein the reflectivityinformation of the traffic infrastructure element is a 3D lightdetection and ranging (LiDAR) point cloud.
 4. The computer-implementedmethod of claim 1, wherein the semantic map information of the trafficinfrastructure element includes previous reflectivity information of thetraffic infrastructure element.
 5. The computer-implemented method ofclaim 1, wherein the providing of the instructions to the autonomousvehicle includes logging a difference between the reflectivityinformation of the traffic infrastructure element and the semantic mapinformation of the traffic infrastructure element, or proceeding asnormal.
 6. The computer-implemented method of claim 1, furthercomprising comparing visual information of the traffic infrastructureelement with the semantic map information of the traffic infrastructureelement.
 7. The computer-implemented method of claim 6, wherein thevisual information of the traffic infrastructure element includes colorinformation of the traffic infrastructure element.
 8. A systemcomprising: one or more processors; and at least one computer-readablestorage medium having stored therein instructions which, when executedby the one or more processors, cause the system to: analyze a trafficinfrastructure element; determine reflectivity information of thetraffic infrastructure element based on the analysis of the trafficinfrastructure element; compare the reflectivity information of thetraffic infrastructure element with semantic map information of thetraffic infrastructure element; and provide instructions to anautonomous vehicle based on the comparison of the reflectivityinformation of the traffic infrastructure element with the semantic mapinformation of the traffic infrastructure element.
 9. The system ofclaim 8, wherein the traffic infrastructure element is a road sign, atraffic light, or signage.
 10. The system of claim 8, wherein thereflectivity information of the traffic infrastructure element is a 3Dlight detection and ranging (LiDAR) point cloud.
 11. The system of claim8, wherein the semantic map information of the traffic infrastructureelement includes previous reflectivity information of the trafficinfrastructure element.
 12. The system of claim 8, wherein theinstructions provided to the autonomous vehicle includes logging adifference between the reflectivity information of the trafficinfrastructure element and the semantic map information of the trafficinfrastructure element, or proceeding as normal.
 13. The system of claim8, wherein the instructions which, when executed by the one or moreprocessors, cause the system to compare visual information of thetraffic infrastructure element with the semantic map information of thetraffic infrastructure element.
 14. The system of claim 13, wherein thevisual information of the traffic infrastructure element includes colorinformation of the traffic infrastructure element.
 15. A non-transitorycomputer-readable storage medium comprising: instructions stored on thenon-transitory computer-readable storage medium, the instructions, whenexecuted by one more processors, cause the one or more processors to:analyze a traffic infrastructure element; determine reflectivityinformation of the traffic infrastructure element based on the analysisof the traffic infrastructure element; compare the reflectivityinformation of the traffic infrastructure element with semantic mapinformation of the traffic infrastructure element; and provideinstructions to an autonomous vehicle based on the comparison of thereflectivity information of the traffic infrastructure element with thesemantic map information of the traffic infrastructure element.
 16. Thenon-transitory computer-readable storage medium of claim 15, wherein thetraffic infrastructure element is a road sign, a traffic light, orsignage.
 17. The non-transitory computer-readable storage medium ofclaim 15, wherein the reflectivity information of the trafficinfrastructure element is a 3D light detection and ranging (LiDAR) pointcloud.
 18. The non-transitory computer-readable storage medium of claim15, wherein the semantic map information of the traffic infrastructureelement includes previous reflectivity information of the trafficinfrastructure element.
 19. The non-transitory computer-readable storagemedium of claim 15, wherein the instructions provided to the autonomousvehicle includes logging a difference between the reflectivityinformation of the traffic infrastructure element and the semantic mapinformation of the traffic infrastructure element, or proceeding asnormal.
 20. The non-transitory computer-readable storage medium of claim15, wherein the instructions, when executed by the one more processors,cause the one or more processors to compare visual information of thetraffic infrastructure element with the semantic map information of thetraffic infrastructure element, wherein the visual information of thetraffic infrastructure element includes color information of the trafficinfrastructure element.